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^ The Differentiated Outcome Hypothesis or 

When Hill We Stop Using Conventional Achievement as the 
y ■ y Sole Criterion for Evaluating Unconventional Instruction 

Bruce H. lUckman 
Rutgers University 

^. The Problem 

It stands to reason that different kinds of interventions and programs 
should result in different outcomes, yet the overriding tendency is to rely 
on standardized achievement data as the basic criteria for evaluation. When 
an "experimental" program falls co produce greater galnb In istandardiiied 
achievement than the "conventional" or "traditional" program, the inevitable 
conclusion drawn is that the experimental program is therefore not worth the 
extra trouble. Perhaps this attitude reflects a tendency by educators, 
parents, and, all too often, evaluators, to rely on a limited number of out- 
come measures - often restricted to the standardised achievement variety* 
Ihis Is defended on the grounds that (a) we get these data anyway (i.e., 
standardized achievement testing Is a part of the regimen of most school 
^ districts); (b) other things cannot be measured as easily nor as well 

(i.e., how do you measure liking for school, creativity, etc.); (c) producing 
achievement of the standardized achievement- test variety is the nitty- gritty 
of education; all the other stuff - e.g., creativity, attitudes, self- 
discipline - are frills. 
2. The Conception 

^ The major issues for measurement specialists and evaluators to con- 

tribute to are twofold: first, what outcomes should you be measuring in 
your evaluation of a program or other form of intervention, and second, 

O 
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how do you measure these outcomes* The Idea that different p rograms when 
successful yield different outcomes peculiar to those i>rograms I call the 
differentiated outcome hypothes is* It may also be applied to Individual 
differences; I.e., different students will maximize performance on different 
outcomes as a function of their characteristic way of relating to their 
environment. 

The differentiated outcomes hypothesis suggests that evaluators consider 
a range of outcomes that may accurately reflect the results of the treatment 
being evaluated. Three trends appear In many evaluations,* namely: (1) the 
Input side Is considerably differentiated. I.e., treatments are described 
In detail - often with respect to some Instructional model, (2) the outcome 
side Is often restricted to achievement measurement alone, (3) little concern 
Is given to Individual differences in students other than to control for 
them. Since achievement measurement has evolved alongside the basic class- 
room, group-oriented instructional procedure which we tend to label "con- 
ventional" or "traditional," it is not unreasonable to expect that the 
conventional instructional approach will maximize achievement. If this 
expectation is correct, then the use of achievement as a sole evaluation 
criterion would favor the conventional or tried- and- true approach to in- 
struction. Such evaluation v7ould have to be considered short- sighted and 
even potentially biased in its limitations. 

The position to be offered in this paper, based on the differentiated 
outcome hypothesis » is that evaluation studies should Include a range of 
outcome measures specifically chosen to sample outcomes that are different 
not only in degree but in kind. 



*It must be emphasized that the majority of educational evaluations are 
unpublished. Published instructional research does not fall into this 
threefold pattern to the same degree as do evaluation studies. 



Some Support 

Gagne^(1971, 1972) has distinguished five domains for classifying learn* 
Ing processes, namely: (1) verbal Information, (2) Intellectual skills, 
(3) cognitive strategies, (4) motor skills, and (5) attitudes. These 
categories can be used as well for classifying outcomes and suggest a range 
of areas In which measurement of program effects might be undertaken, Glaser 
(1972 p. 12) also suggests measurement outside the "narrow band cf tradi- 
tional academic outcomes,'' with the Inclusion of ''measures of process and 
style, of cogniclve and non-cognltlve development, and of performance In 
more natural settings." He also argues strongly in favor of the Inclusion 
of Individual difference measures of cognitive process within Instrucclonal 
research, Lohnes (1973) while arguing for the Inclusion of Intellectual 
development as an Important outcome of many (possibly all) Instructional 
treatments, cites arguements for the Inclusion of differentiated outcomes. 
Tyler (1931) said quite succinctly: 

"Any learning situation has multiple outcomes. While 
the child is acquiring information, knowledges, and 
skills, there is also taking place concomitant learnings 
in attitudes, appreciations, and interests. This v±m 
indicates a shift from a narrow conception of subject: «- 
matter outcomes to a broader conception of growth and 
development of individuals." (p. 48). 

And finally, Croubach (1971, p. 460) states: "Consequently, an Ideally 

suitable battery for evaluation purposes will include separate meai^ures 

of all outcomes the users of the information consider important." 

The Model 

I would like to suggest five categories or classes of outcomes, based 
on pronouncements like those above, that might reasonably form the basis 
for the application of the differentiated outcome hypothesis and which 
are represented In Figure 1, These are: (a) specific knowledge and 



A. SPECIFIC KNOWLEDGE AND COMPREHENSION (SPEC COG) 

Subject-matter achievement (e.g., knowledge and understanding of 
math, science, language arts, social studies, machine operation, 
etc., based on specific curriculum content and specific objectives 
of program. ) 

B. GENERAL KNOlfLEDGE AND COMHIEHENSION (GEN COG) 

intelligence 
aptitudes 
general abilities 
reading ability 
reasoning ability 
knowledge of culture 

C. THINKnJG AND HlOBLEM- SOLVING (HIGH COG) 

(these characteristics do not typically have names and may not be 
measured in typical test situations) 
analysis 
synthesis 
evaluation 
dec is ion<*maIclng 

D. ATTITODES AND VALUES (AFF) 

attitudes toward school 
attitudes toward self 

attitudes toward intemal*extemal control 

vocational maturity 

interests 

prejudice 

tolerance of ambiguity 
value priority 

E. LEARNING-RELATED BEHAVIOR (BEH) 

attendance 

time devoted to learning or problem- solving 

self-discipline 

initiative 

cooperativeness 

performance (in an actual or simulated setting; not paper- and- pencil) 



Figure 1 Categories of Differentiated Outcomes and Examples Within Each 



comprehension, (b) general knovjledge and comprehension, (c) thinking and 
problem- solving, (d) attitudes and values, and (e) learning* related 
behavior. Figure 1 lists some possible outcomes in each category in order 
to make these categories clearer In meaning. (No attempt has been made to 
be exhaustive. ) 

It is recocanended that at least one and preferably more than one 
variable in each category be Included in every sunuaatlve evaluation to at 
least broaden our knowledge of the differential effects of different educa- 
tional Interactions. The specific variables chose n for measurement should 
be based ^ insofar as possible , on tha objectives of each of the t'jreatnaents 
being compared . Unlike Ebel (1973), my recommendation is that program 
effects be measured in terms of mo7:e than just specific knowledge and com* 
prehension goals and thus include objectives In all five areas. (It may be 
necessary first to get program designers to generate objectives in all five 
areas; othen'7ise some areas will continue to be typically overlooked.) 
Some Current Evidence 

It would be useful at this point to describe some of the work that has 
been completed using the notion of differentiated outcomes. While there 
may be a number of suitable examples in the literature, only a few will 
be described * all but one of which were done by my students and myself. 
That one was done by Worthen (1968) in an effort to compare two methods of 
task presentation In mathematics, the discovery approach versus the ex* 
pository approach. I will not go into any detail regarding these Inputs 
but will attend primarily to his differentiated outcomes. Five outcome 
variables x^ere measured. These were (1) initial concept learning, (2) re- 
tention of concepts, (3) transfer of concepts, (4) transfer of heuristics 
(or strategies), and (5) attitudes to\7ard content. The first variable 



fits in^o the SPEC COG (see Figure 1) category, the fourth Into the HIGH 
COG category, and the fifth Into AFP, Variable 3 could be either SPEC COG 
or HIQl COG v;hlle variable 2 does not fit the categorization. Perhaps 
retention measures fit LFBC COG or perhaps they require a category of their 
own. Worthen found the two approaches to task presentation not to differ 
significantly on the Initial learning (#1) or attitudes (fi5) they produced; 
that the discovery group significantly exceeded the expository group on 
retention (#2) and transfer of heuristics (iS^4); and that the discovery group 
exceeded the expository group on transfer of concepts (#3) with the dif- 
ference approaching significance. Overall, let us sunmiarize by saying that 
differences V7ere found mainly on HIGH COG variables as contrasted to SPEC 
COG cr AFF variables. 

Let me describe some of my studies so that a pattern may become 
detectable. Tuclunan and Oreflce (1973) compared four different procedures 
for teaching first year community college students a unit in accounting. 
These V7ere: (a) self- instruction via tapes and booklets, (b) programmed 
Instruction X7ithin a classroom setting, (c) progranoned instruction and 
lectures within a classroom, and (d) traditional lecture - discussion in- 
struction. As one proceeds from treatments (a) to (d), instruction is seen 
as being decreaslngly student- initiated and increasingly teacher- initiated. 
Also, within this study, students V7ere classified as either abstract or 
concrete in personality orientation. Measured outcomes Included: (1) achieve- 
ment of accounting objectives (SPEC COG), (2) student time spent in the 
instructional process (BEH), and (3) attitudes tm^ard the form of instruc- 
tion received (AFF)» No differences between treatments, beti^een personal- 
ities » or V7lthin their Interaction was found on the achievement (SPEC COG) 
variable. On the attitude (AFF) measure, the self-study approach V7as found 



to be clearly preferred and the lecture-* progrananed textbook approach liked 
least but even oiore strlklns was the interaction betitireen treatment and 
personality. Abstract students disliked the lecture-prograxrmed textbook 
most among the four treatment: while concrete students preferred this 
approach; exactly the reverse v;as true for the programmed textbook without 
lecture approach: abstract students preferred it and concrete students 
disliked it. Consistent with their preference patterns, concrete students 
registered their dislike for the programmed approach alone by spending 
noticeably less time on it than on any of the other approaches. 

Again, V7e find no differences on the SPEC COG variable with differences 
appearing this time on AFF and BEH measures, ^k>reover, our differentiated 
outcomes are sensitive not only to treatment differences but to personality 
differences in combination with treatments as well (what Cronbach and Gleser^ 
1965, call aptitude- treatment interactions and Hunt, 1971, calls matching 
models). 

The missing element in the above study seemed to be a HIGH COG measure 
and so another study V7as done (Elliott and IXickman, 1973) comparing indi-* 
vidualized and traditional accounting instruction - this time in a 
naturalistically occurring way. A completely individualized program was 
compared to an entirely classroom*based one on a series of differentiated 
outcome measures including: (1) achievement on accounting objectives 
(SPEC COG), (2) student instructional time (BEH), (3) Student attitudes 
toward school (AFF), and (4) a problem- solving measure that included 
measures of knowledge (SPEC COG), search initiative, time spent and 
solution length (BEHs), and adequacy of problem solutions (HIGH COG). 

On the achievement measure (SPEC COG), again no treatment differences 
were found. No differences were found on the attitude (AFF) measure as 
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well. Individualized students were able to complete Instruction In less time 
than classroom"-! aught students (the BEH measure).* 

Let us focus our attention on the multl«- faceted problem- solving measure. 
This was a take home problem In an aspect of accounting not coveted in either 
treatment, viz., taxes on Investments* Thus, It was unfamiliar In Its 
specifics but within the range of subject*- matter. It contained within it 
some unfamilia'* terms. Students were asked to (1) provide definitions for 
the unfamiliar terms (SPEC COG), (2) indicate as many solutions as they could, 
in writing^ for the problem (HIGH COG), and (3) keep track of and report time 
spent working on the problem (BEH). They were told to use whatever resources 
they chose to help them solve the problem but that they must (4) report the 
resources used (BEH). Finally, (5) problem- solution length was examined (BEH). 

You can probably guess what happened. On the SPEC COG measure, no treat- 
ment differences were found. On the problem solutions (HIGH COG) as judged by 
accountants (with adequate scoring reliability demonstrated), individualized 
studerxs did significantly better than classroom- taught students. Higher 
scores on each of the behavior measures were also obtained by individualized 
students. 

So far we have reported on discovery and individualised instruction 
compared to more conventional approaches with the general finding being no 
differences in specific cognitive achievement between approaches but dif- 
ferences favoring one approach over the other in higher cognitive processes. 
Let us look closely at one more study. Tuckman, Cochran, & Travers (1974) 
compared open classroom instruction to conventional instruction at the 
elementary school level in terms of the following measures from students; 

^^Personality V7as alscr included in this study but produced no significant 
main effects or interactions. 



(1) achievement In language arts, reading, and math (SPEC COG), (2) claa- 
alflcatlon task skills (HIGH COG), (3) attitudes toward school (AFF), and 
(4) attitudes toward self (AFF). Significant differences were found only 
on the two affective measures *- In both cases In favor of the open class- 
room group. 

Although the above finding somewhat breaks the pattern. It can be seen 
as consistent with the treatments and hence In support of the differentiated 
outcome hypothesis. The open classroom program studied had Just completed 
Its first year and other findings In the study* showed that teacher social 
behavior was perhaps the most noticeable area of Instructional change 
provoked by the short amount of In- service training received by teachers. 
Basic curriculum changes had not yet occurred. In turn, teacher behavior 
characterized more by warmth and acceptance resulted, not unreasonably. In 
affective gains among students without concomitant Improvement in the hlgiier 
cognitive area. Of course, by now the no- differences- In- achievement finding 
Is also becoming a familiar outcome. 
More to Come 

Some other outcomes currently being (cr recently having been) measured 
In our work include: student alienation (in modular- scheduled and con- 
ventional middle schools), attitudes about internal- external (fate) control 
(as the result of a high school political education program), teacher 
ratings of student behavior (in a Junior high pre- vocational program for 
problem students), tolerance of ambiguity (in a high school senior year 
alternative program as contrasted to a conventional program) ^ life skills • 
health, money management, and family life self- reported behaviors (among 

*A number of measures of teacher behavior were also included in this study. 
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Puerto Rican and non Puerto Rlcan students), reasoning ability (among students 
in a school district*wide program using instruction based on behavioral ob- 
jectives), and ability to complete an unfamiliar experiment (among students 
having access on a self* initiated basis or those not having this access to 
single-concept filmstrips). In each instance, In addition to the outcomes 
mentioned, others are chosen in an attempt to cover all five categories of 
the model. 
7. The Moral 

The differentiated outcome hypothesis is probably more a mandate or a 
proposition cf good sense than a hypothesis. It is the function of a sum- 
matlve evaluation to demonstrate those outcomes that a program can achieve 
better than its alternatives In addition to any that it cannot achieve as well. 
It Is then the province of those-v/ho-provide or those- that-experlence or those- 
that- finance the program to decide whether its outcomes are a sufficient basis 
for Its continuation. However, It must be emphasized that all programs, even 
those that are considered alternatives to one another, cannot be expected to 
yield the same outcomes. One should be able to predict the kinds of outcomes 
that a program will maximize (If done optimally) by examining Itc goals and 
procedures. The job of the evaluator Is not to decide which outcomes are the 
most Important but to demonstrate which outcomes, if any, are maximized. While 
those who theorize about learning and instruction typically concentrate on 
educational Inputs, It Is left to evaluators to be concerned about outcomes.* 
And since the list of possible outcome measures is endless, the evaluator 
must have a basis for choosing the ones to examine. Instead of accepting con- 
ventional achievement test scores as the "shoe" and maintaining "if the shoe 
fits, wear It,'' evaluators must be willing and able to examine the full range 
of "sizes" and "styles." 

^Although as Lohnes (1973 suggests, evaluators must also determine that the In- 
puts or processes are consistent with the name that has been used to label them. 
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